An Information Theoretic Approach to Managing Multiple Decision Makers

Citizen science and human computation involves working with multiple, untrusted decision makers, whose performance depends on training, rewards, ability and interest. We first present methods for screening workers and selecting informative objects to label. We then demonstrate Bayesian Classifier Combination as an effective method for classifying documents using unreliable crowdsourced labels. Finally, we explain how the Bayesian Classifier Combination model could form the basis of a single information-theoretic framework for screening workers, selecting documents and training tasks and determining rewards.